OS design for non-cache-coherent systems

Which operating system structures are appropriate for machines with either no cache coherence, or else a number of distinct “coherence islands”? While a clear consensus on the architecture of future multicore processors has yet to emerge, it seems that machines without system-wide cache coherence are likely. Some research chips, like the Intel Single-Chip Cloud computer [3] and the Beehive computer [5], have already emerged. This poster presents early work exploring the tradeoffs which characterize the OS design space for such machines which share system memory, but do not provide cache coherence across the whole machine. The trend driving the re-emergence of non-cachecoherent systems in general-purpose computer architecture is the quest for greater scalability from multi-core processors, and the bottlenecks inherent in a globallycoherent shared-memory model [4,7]. Now is the time to consider what the system software stack must look like on machines like this. As Baumann et al. [2] (among others) have pointed out, modern OSes like Windows and Linux are essentially large, multithreaded programs which assume coherent shared memory, and then apply performance optimizations based on heuristics of the underlying memory system. This model by itself is not going to work on a non-coherent machine. There are several alternative OS models, which range from carefully rewriting a shared-memory kernel to include explicit cache-flush operations, through refactoring the OS to treat all shared memory with release semantics, eschewing sharing among coherence islands and relying on message passing, all the way to running a separate instance of a conventional OS on each core and programming the machine as if it was a cluster (cluster-on-chip). Simply refactoring a shared-memory OS to explicitly flush caches will not provide any scalability benefits over hardware-coherent systems without replacing critical data structures with ones that induce less cache traffic. Cache